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DETAILED ACTION 
Response to Amendment 

Claims 1-37,39-42 are pending in the application. Claims 1-37,39-42 were amended 
claim 38 has been cancelled. This Action is Final. 

Claim Objections 

The numbering of claims is not in accordance with 37 CFR 1 .126 which requires 
the original numbering of the claims to be preserved throughout the prosecution. When 
claims are canceled, the remaining claims must not be renumbered. When new claims 
are presented, they must be numbered consecutively beginning with the number next 
following the highest numbered claims previously presented (whether entered or not). 

Misnumbered claim 37 should been renumbered claim 38. 

Claim Rejections - 35 USC § 102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

2. Claims 35-37,39 are rejected under 35 U.S.C. 102(b) as being anticipated by Lin 
("Lin", An Ink and Voice Annotation System for DENIM). 

As per independent claim 35, Lin discloses a computer implemented method for 
displaying objects with annotations, the method comprising the steps of: retrieving an 



Application/Control Number: 10/043,575 Page 3 

Art Unit: 2174 

image (Page 3 Figure 4a); displaying the image with a visual notation that an annotation 
exists (Page 3 Figure 4c item 1 ); receiving user selection of an image (Page 3 Figure 4a 
selected from Page 2 Figure 2); and outputting a notation associated with the selected 
image (Page 3 Figure 4b and 4c) determining whether the annotation includes text; 
retrieving a text annotation for the selected image; and displaying the retrieved text with 
the image (Page 5 lines 16-26). 

As per claim 36, which is dependent on claim 35, Lin discloses a method wherein 
the annotation is text and the step of outputting is displaying the text proximate an 
image that it annotates (Page 3 Figure 3). 

As per claim 37, which is dependent on claim 35, Lin discloses a method wherein 
the annotation is an audio signal and the step of outputting is playing the audio signal 
(Page 3 Figure 4b). 

As per claim 39, which is dependent on claim 35, Lin discloses a method further 
comprising the steps of: determining whether the annotation includes an audio signal 
(Page 3 Figure 4c item 1 ); retrieving a audio signal for the selected image (Page 3 lines 
16-17); and wherein the step of outputting is playing the audio signal (Page 3 Figure 
4b). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
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invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-34,40 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Anderson ("Anderson", US 6,499,016) in view of Balabanovic ("Balabanovic", 
Multimedia Chronicles for Business Communication). 

As per independent claim 1 , Anderson discloses an apparatus for direct 
annotation of objects, the apparatus comprising: a display device for displaying one or 
more images (Column 2 lines 56-61); an audio input device for receiving an audio input 
(Column 2 lines 56-61); and a direct annotation creation module coupled to receive an 
input audio signal from the audio input device and to receive a reference to an image 
from the display device, the direct annotation creation module creating an annotation 
object that associates an input audio signal with an image (Column 3 lines 13-17). 
Anderson fails to distinctly point out automatically creating an annotation object 
independent from the image. However, Balabanovic teaches in response to receiving 
the audio input signal and the reference to the image, automatically creating an 
annotation object, independent from the image, that associates the input audio signal 
with the image (Column 7 lines 34-48, lines 51-61). Therefore it would have been 
obvious to an artisan at the time of the invention to combine the teaching of Balabanovic 
into the system of Anderson. Motivation to do so would have been to provide a user with 
a freeform surface which audio clips can be arranged. 

As per claim 2, which is dependent on claim 1 , Anderson-Balabanovic discloses 
the apparatus of claim 1 further comprising an annotation display module (Anderson, 
Column 3 lines 19-22) coupled to the direct annotation creation module, the annotation 
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display module generating symbols or text representing the annotation objects 
(Anderson, Column 3 lines 13-17; corresponding text annotations). 

As per claim 3, which is dependent on claim 1, Anderson-Balabanovic discloses 
an annotation audio output module coupled to the direct annotation creation module, the 
annotation audio output module generating audio output in response to user selection of 
an annotation symbol representing an annotation object (Balabanovic, Column 7 lines 
50-62, Column 8 lines 23-27). 

As per claim 4, which is dependent on claim 1 , Anderson-Balabanovic discloses 
The apparatus of claim 1 further comprising: an audio vocabulary storage for storing a 
plurality of audio signals and corresponding text strings (Anderson, Figure 3a); an audio 
vocabulary comparison module coupled to the audio input device (Anderson, Column 3 
lines 66-67), the audio vocabulary storage and the direct annotation creation module 
(Anderson, Figure 1), the audio vocabulary comparison module receiving audio input 
and finding a corresponding text string that matches the audio input (Anderson, Column 
5 lines 31-35); and wherein the direct annotation creation module uses text strings 
found by the audio vocabulary comparison module to create the audio annotation 
(Anderson, Column 5 lines 52-55). 

As per claim 5, which is dependent on claim 1 , Anderson-Balabanovic discloses 
the apparatus further comprising: an audio vocabulary storage for storing a plurality of 
audio signals and corresponding text strings (Anderson, Figure 1); a dynamic 
vocabulary updating module coupled to the audio vocabulary storage and the audio 
input device (Anderson, Figure 1), the dynamic vocabulary updating module for 
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displaying an interface to create a new entry in the audio vocabulary storage (Anderson, 
Column 5 lines 38-46), the dynamic vocabulary updating module receiving an audio 
input and a text string and creating the new entry in the audio vocabulary storage 
(Anderson, Column 5 lines 50-51). 

As per claim 6, which is dependent on claim 1 , Anderson-Balabanovic discloses 
the apparatus of claim 1 further comprising a media object cache for storing media and 
annotation objects (Anderson, Column 4 lines 61-67). 

As per independent claim 7, Anderson-Balabanovic discloses an apparatus for 
use with a system for storing, accessing, and presenting objects such as video objects, 
text objects, audio objects, or image objects, direct annotation of objects, the apparatus 
comprising: a direct annotation creation module coupled to receive an input audio signal 
and reference to an image (Anderson, Column 4 lines 61-63), the direct annotation 
creation module creating an annotation object, independent of the image (Balabanovic, 
Column 7 lines 51-61), that associates a symbol or text with the image (Anderson, 
Column5 lines 31-35); and an annotation display module coupled to the direct 
annotation creation module (Anderson, Figure 1), the annotation display module 
generating the symbol or text representing the annotation object on a display device 
(Anderson, Column 6 lines 26-31,35-38). 

As per independent claim 8, Anderson discloses an apparatus for direct 
annotation of objects for use with a system for storing, accessing, and presenting 
objects such as video objects, text objects, audio objects, or image objects, the 
apparatus comprising: a direct annotation creation module coupled to receive an input 
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audio signal and a reference to an image (Anderson, Column 4 lines 61-63), the direct 
annotation creation module creating an annotation object, independent of the image 
(Balabanovic, Column 7 lines 51-61), that associates the input audio signal and the 
image (Anderson, Column 5 lines 31-35), the annotation object including at least an 
audio input field, an image reference field, and an annotation location field 
(Balabanovic, Column 7 lines 34-61), the annotation audio output module coupled to the 
direct annotation creation module, the annotation audio output module generating audio 
output in response to user selection of an annotation symbol representing the 
annotation object (Balabanovic, Column 7 lines 50-62, Column 8 lines 23-27). 

As per independent claim 9, Anderson-Balabanovic discloses an apparatus for 
direct annotation of objects, the apparatus comprising: a media object storage for 
storing media and annotation objects (Anderson, Figure 1 item 24); and a direct 
annotation creation module coupled to receive an input audio signal and a reference to 
an image (Anderson, Column 4 lines 61-63), the direct annotation creation module 
creating an annotation object, independent of the image (Balabanovic, Column 7 lines 
51-61), that associates the input audio signal and the image (Anderson, Column 5 lines 
31-35), the direct annotation creation module storing the audio annotation in the media 
object storage (Anderson, Column 5 lines 52-57;May be deleted).. 

As per independent claim 10, Anderson-Balabanovic discloses a computer 
implemented method for direct annotation of objects, the method comprising the steps 
of: displaying an image (Anderson, Figure 1 item 18); receiving audio input (Anderson, 
Column 3 lines 10-13); detecting selection of an image (Anderson, Column 3 lines 13- 
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17); and creating an annotation object, independent of the selected image (Balabanovic, 
Column 7 lines 51-61), between the selected image and the audio input (Anderson, 
Column 3 lines 13-17)., the annotation object including at least an audio input field, an 
image reference field, and an annotation location field (Balabanovic, Column 7 lines 34- 
61). 

As per claim 1 1 , which is dependent on claim 1 0, Anderson-Balabanovic 
discloses a method where the step of displaying is performed before or simultaneously 
with the step of receiving (Anderson, Column 3 lines 10-16; recording at the time of 
capture). 

As per claim 12, which is dependent on claim 10, Anderson-Balabanovic 
discloses a method wherein the step of receiving is performed before or simultaneously 
with the step of displaying (Anderson, Column 3 lines 10-16; recording at the time of 
capture). 

As per claim 13, which is dependent on claim 10, Anderson-Balabanovic teaches 
a method wherein the step of detecting selection includes detecting a portion of the 
image (Balabanovic, Column 8 lines 49-51); and wherein the annotation creates an 
association between the portion of the image and the audio input (Balabanovic, Column 
8 lines 49-51, Figure 4). 

As per claim 14, which is dependent on claim 10, Anderson-Balabanovic teaches 
a method further comprising the step of displaying a visual notation that the image has 
an annotation (Balabanovic, Figure 4; apparent that the document has an annotation by 
displaying the image and the clip). 
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As per claim 15, which is dependent on claim 14, Anderson-Balabanovic teaches 
the visual notation being text or a symbol (Figure 4; picture of man's face). 

As per claim 16, which is dependent on claim 10, Anderson-Balabanovic 
discloses a method wherein the step of creating an annotation includes creating an 
annotation object and storing the annotation object in an object storage (Anderson, Fig 
3a). 

As per claim 17, which is dependent on claim 10, Anderson-Balabanovic 
discloses a method further comprising the step of recording the audio input received 
(Anderson, Column 3 lines 10-13). 

As per claim 18, which is dependent on claim 17, Anderson-Balabanovic 
discloses a method wherein the step of creating an annotation includes creating an 
annotation object and storing the recorded audio input as part of the annotation object 
(Anderson, Column 3 lines 39-43). 

As per claim 19, which is dependent on claim 10, Anderson-Balabanovic 
discloses a method, further comprising the step of comparing the audio input to a 
vocabulary to produce text (Anderson, Column 3 lines 59-65). 

As per claim 20, which is dependent on claim 19, Anderson-Balabanovic 
discloses a method, wherein the step of creating an annotation includes creating an 
annotation object and storing the text as part of the annotation object (Anderson, 
Column 5 lines 31-36). 
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As per claim 21 , which is dependent on claim 10, Anderson-Balabanovic 
discloses a method further comprising the steps of comparing the audio input to a 
vocabulary (Anderson, Column 5 lines 31-35); determining if the audio input has a 
matching entry in the vocabulary (Anderson, Column 5 lines 36-38); and storing the 
entry as part of the annotation object if the audio input has a matching entry in the 
vocabulary (Anderson, Column 5 lines 52-57). 

As per claim 22, which is dependent on claim 21, Anderson-Balabanovic 
discloses a method further comprising the steps of: determining if the audio input has a 
close match in the vocabulary (Anderson, Column 5 lines 36-38); displaying the close 
matches (Anderson, Column 5 lines 40-46 \options)\ receiving input selecting a close 
match (Anderson, Column 5 lines 40-46 ;opf/ons chosen)] and storing the selected close 
match as part of the annotation object if the audio input has a close match in the 
vocabulary (Anderson, Column 5 lines 52-57). 

As per claim 23, which is dependent on claim 22, Anderson-Balabanovic 
discloses the method, further comprising the step of displaying a message that the 
image has not been annotated if there is neither a matching entry in the vocabulary nor 
a close match in the vocabulary (Anderson, Column 5 lines 47-51). 

As per claim 24, which is dependent on claim 22, Anderson-Balabanovic 
discloses a method, further comprising the following steps if there is neither a matching 
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entry in the vocabulary nor a close match in the vocabulary: receiving text input 
corresponding to the audio input (Anderson, Column 5 lines 47-51 \email for instant 
messages is text): updating the vocabulary with a new entry including the audio input 
and the text input (Anderson, Column 5 lines 52-57 after corrected)) and wherein the 
received text is stored as part of the annotation object (Anderson, Column 5 lines 52-57; 
after corrected). 

As per claim 25, which is dependent on claim 10, Anderson-Balabanovic 
discloses a method, further comprising the steps of: receiving text input corresponding 
to the audio input (Anderson, Column 5 lines 47-51 ); updating the vocabulary with a 
new entry including the audio input and the text input (Anderson, Column 5 lines 52-57). 

As per independent claim 26, Anderson discloses a method for direct annotation 
of objects, the method comprising the steps of: displaying an image (Anderson, Figure 1 
item 18); receiving audio input (Anderson, Column 3 lines 10-13); detecting selection of 
an image (Anderson, Column 3 lines 13-17); comparing the audio input to a vocabulary 
to produce text (Anderson, Column 3 lines 13-16); and creating an annotation between 
the selected image and the text (Anderson, Column 3 lines 13-16). 

As per claim 27, which is dependent on claim 26, Anderson-Balabanovic 
discloses further comprising the step of recording the audio input received (Anderson, 
Column 3 lines 39-42). 

As per claim 28, which is dependent on claim 27, Anderson-Balabanovic 
discloses the method, wherein the step of creating an annotation includes creating an 
annotation object including a reference to the selected image, the recorded audio input 
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and the text (Figure 3a), and storing the annotation object in an object storage 
(Anderson, Figure 1 item 24). 

As per claim 29, which is dependent on claim 26, Anderson-Balabanovic 
discloses the method, wherein the step of creating an annotation includes creating an 
annotation object and storing the text as part of the annotation object (Anderson, 
Column 3 lines 30-39). 

As per claim 30, which is dependent on claim 26, Anderson-Balabanovic 
discloses a method further comprising the steps of determining if the audio input has a 
matching entry in the vocabulary (Anderson, Column 5 lines 56-41); and storing the 
entry as part of the annotation object if the audio input has a matching entry in the 
vocabulary (Anderson, Column 5 lines 52-57). 

As per claim 31 , which is dependent on claim 29, Anderson-Balabanovic 
discloses a method, further comprising the steps of: determining if the audio input has a 
close match in the vocabulary (Anderson, Column 5 lines 36-41); displaying the close 
matches (Anderson, Column 5 lines 38-46 .options)] receiving input selecting a close 
match (Anderson, Column 5 lines 38-46; chosen options)] and storing the selected 
close match as part of the annotation object if the audio input has a close match in the 
vocabulary (Anderson, Column 5 lines 52-57). 

As per claim 32, which is dependent on claim 30, Anderson-Balabanovic 
discloses the method, further comprising the step of displaying a message that the 
image has not been annotated if there is neither a matching entry in the vocabulary nor 
a close match in the vocabulary (Anderson, Column 5 lines 46-51). 
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As per claim 33, which is dependent on claim 30, Andersbn-Balabanovic 
discloses a method, further comprising the following steps if there is neither a matching 
entry in the vocabulary nor a close match in the vocabulary: receiving text input 
corresponding to the audio input (Anderson, Column 5 lines 46-51); updating the 
vocabulary with a new entry including the audio input and the text input (Anderson, 
Column 5 lines 50-51); and wherein the received text is stored as part of the annotation 
object (Anderson, Column 5 lines 52-57). 

As per claim 34, which is dependent on claim 26, Anderson-Balabanovic 
discloses a method, further comprising the steps of: receiving text input corresponding 
to the audio input (Anderson, Column 46-51 ); updating the vocabulary with a new entry 
including the audio input and the text input (Anderson, Column 5 lines 50-51). 

As per independent claim 40, Anderson-Balabanovic discloses a method for 
retrieving images, the method comprising the steps of: receiving audio input (Anderson, 
Column 6 lines 9-11); determining annotation objects that reference a close match to 
the audio input (Anderson, Column 6 lines 34-42); retrieving the images that are 
referenced by the determined annotation objects (Anderson, Column 6 lines 41-42); and 
displaying the retrieved images (Anderson, Column 6 lines 34-42). 

4. Claims 41 and 42 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Anderson ("Anderson", US 6,499,016) and Balabanovic ("Balabanovic", Multimedia 
Chronicles for Business Communication). 
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As per claim 41 , which is dependent on claim 40, Anderson-Balabanovic 
discloses a method wherein the step of determining annotation objects further 
comprising the steps of: comparing the audio input to an audio signal reference by an 
annotation object (Column 5 lines 30-35); and determining a close match between the 
audio input to the audio signal reference by an annotation object if a probability metric is 
greater than an a threshold (Column 5 lines 35-38). Anderson-Balabanovic fails to 
disclose a threshold of 80%. However, Official Notice is taken that a threshold of 80% is 
well known in the art, 80% is not a definitive threshold, and could be replaced by any 
other value. Therefore it would have been obvious to combine the method of Anderson- 
Balabanovic with the current teaching. Motivation to do so would have been to provide a 
standard of matching. 

As per claim 42, which is dependent on claim 40, Anderson-Balabanovic 
discloses a method wherein the step of determining annotation objects further 
comprising the steps of: determining the annotation objects for a plurality of images; for 
each annotation object, comparing the audio input to an audio signal reference by an 
annotation object (Column 5 lines 30-35); and determining a close match between the 
audio input to the audio signal reference by an annotation object if a probability metric is 
greater than an a threshold (Column 5 lines 35-38). Anderson-Balabanovic fails to 
disclose a threshold of 80%. However, Official Notice is taken that a threshold of 80% is 
well known in the art, 80% is not a definitive threshold, and could be replaced by any 
other value. Therefore it would have been obvious to combine the method of Anderson- 
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Balabanovic with the current teaching. Motivation to do so would have been to provide a 
standard of matching. 

Response to Arguments 

Applicant's arguments with respect to claims 1-36, and 38-41 have been 
considered but are moot in view of the new ground(s) of rejection. 

Furthermore, the Office notes that applicant did not contest the factual assertion 
set forth under Official Notice in claims 41 and 42 on pages 14-15 of the Office Action of 
12/30/2004. 

Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan F Pitaro whose telephone number is 571-272- 
4071 . The examiner can normally be reached on 7:00am - 4:30pm M-Th, and 
alternating F. 

If attempts to reach the examiner by telephone are unsuccessful, the examinees 
supervisor, Kristine Kincaid can be reached on 571-272-4063. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Ryan Pitaro 
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